Determining a location at which a given feature is represented in medical imaging data

ABSTRACT

A computer implemented method and apparatus for determining a location at which a given feature is represented in medical imaging data is disclosed. A first descriptor for a first location in first medical imaging data is obtained. The first location is the location within the first medical imaging data at which the given feature is represented. A second descriptor for each of a plurality of candidate second locations in second medical imaging data is obtained. A similarity metric indicating a degree of similarity with the first descriptor is calculated for each of the plurality of candidate second locations. A candidate second location is selected from among the plurality of candidate second locations based on the calculated similarity metrics. The location at which the given feature is represented in the second medical imaging data is determined based on the selected candidate second location.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority from European PatentApplication No. 21183428.8, filed on Jul. 2, 2021, the contents of whichare incorporated by reference.

TECHNICAL FIELD

The present framework relates to a method and apparatus for determininga location at which a given feature is represented in medical imagingdata.

BACKGROUND

Comparison of medical images occurs frequently in clinical settings. Forexample, in clinical decision making, the progression of a patient'sdisease over time can be as, if not more, important than the currentstatus of that disease. In order to help assess the progression of apatient's disease, medical imaging, e.g., radiology, of a region of thepatient's body including the disease may be performed at differentpoints in time, for example several times over the course of a year. Theresulting images may be compared, for example by a physician, in orderto track the disease over time and hence assess the progression of thedisease.

However, differences in the images, for example differences resultingfrom the precise positioning of the patient relative to the imagingequipment when the images are captured, and/or the imaging modality orprotocols according to which the images are captured, can make itdifficult to locate a given feature (e.g., a particular tumor)represented in one image, in other ones of the images.

Image registration and landmark detection are existing techniques thatcan help address this problem by attempting to spatially align differentimages (or imaging data sets) with one another.

In image registration techniques, different sets of imaging data aretransformed into one coordinate system. In a known image registrationtechnique, a cost function for a given voxel-to-voxel mapping of oneimage to another image in voxel space is calculated, and the mapping isadjusted so as to minimize the cost function. As a result, the locationof a given feature represented by the two images should be the same inthe common coordinate system. However, this technique has drawbacks. Forexample, the ability of this technique to perform accurately and/orreliably is typically limited to cases where the two images both, as awhole, represent the same part or very similar parts of the patient'sbody and/or cases where the same or similar imaging modality or protocolis used. Moreover, as it is based on a voxel-to-voxel mapping of oneimage to another image, this technique is computationally demanding, andhence has a limited ability to provide real-time or near real timeresults without extensive pre-processing of the imaging data and/orwithout use of large computational resources.

In landmark detection techniques, a landmark detector first identifieswell known locations (“landmarks”) in the body by applying trainedclassifiers to both one image and another image. The identifiedlandmarks are mapped between the two images. The images can be broadlyspatially aligned according to the landmarks, and as a result a givenfeature represented in the two images should also be broadly aligned.However, this technique also suffers from drawbacks. For example, itrelies on the images containing landmarks that can be detected by thetrained classifiers. Moreover, the training of the classifiers toidentify landmarks in images can be computationally demanding, andrequires expert annotation of a large number of images to form atraining data set, which is time consuming.

It would be desirable to provide a technique for determining thelocation at which a given feature is represented in medical imagingdata, but which mitigates at least some of the drawbacks of the priorart.

SUMMARY

According to one aspect, there is provided a computer implemented methodof determining a location at which a given feature is represented inmedical imaging data. The medical imaging data includes an array ofelements each having a value. A first descriptor for a first location infirst medical imaging data is obtained, the first location being thelocation within the first medical imaging data at which the givenfeature is represented, the first descriptor being representative ofvalues of elements of the first medical imaging data located relative tothe first location according to a first predefined pattern. A seconddescriptor for each of a plurality of candidate second locations insecond medical imaging data is obtained, each second descriptor beingrepresentative of values of elements of the second medical imaging datalocated relative to the respective candidate second location accordingto the first predefined pattern. For each of the plurality of candidatesecond locations, a similarity metric is calculated. The similaritymetric indicates a degree of similarity between the first descriptor andthe second descriptor for the candidate second location. A candidatesecond location is selected from among the plurality of candidate secondlocations based on the calculated similarity metrics. The location atwhich the given feature is represented in the second medical imagingdata is determined based on the selected candidate second location.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is flow diagram illustrating a method according to an example;

FIG. 2 is a diagram illustrating first medical imaging data according toan example;

FIG. 3 is a diagram illustrating second medical imaging data accordingto an example;

FIG. 4 is a diagram illustrating second medical imaging data accordingto another example;

FIG. 5 is a diagram illustrating output data according to an example;

FIG. 6 is a diagram illustrating first medical imaging data according toan example;

FIG. 7 is a diagram illustrating output data according to anotherexample;

FIG. 8 is a graph a plot of distance against sensitivity for the methoddisclosed herein according to examples as well as comparative methods;

FIG. 9 is a diagram illustrating an apparatus according to an example;

FIG. 10 is a diagram illustrating first/second medical imaging dataaccording to a further example;

FIG. 11 is a diagram illustrating first/second medical imaging dataaccording to a further example;

FIG. 12 is a diagram illustrating first/second descriptors according toa further example;

FIG. 13 is flow diagram illustrating a method according to a furtherexample;

FIG. 14 is flow diagram illustrating a method according to a furtherexample; and

FIG. 15 is a diagram illustrating a workflow according to an example.

DETAILED DESCRIPTION

Referring to FIG. 1 , there is illustrated a computer implemented methodof determining a location at which a given feature is represented inmedical imaging data.

Representations of example medical imaging data with which the methodmay be used are illustrated in FIGS. 2 to 5 . Medical imaging data maybe that captured from performing medical imaging on a patient, forexample Computed Tomography (CT), Magnetic Resonance Imaging (MRI),X-ray, or other imaging techniques. FIG. 2 illustrates a representationof first medical imaging data 220. FIGS. 3 to 5 each illustrate arepresentation of second medical imaging data 330. In each case, themedical imaging data comprises an array of elements each having a value.For example, the medical imaging data may comprise a 2-Dimensional arrayof pixels, each pixel having at least one value. As another example, themedical imaging data may comprise a 3-Dimensional array of voxels, eachvoxel having at least one value. The at least one value may correspondto or otherwise be representative of an output signal of the medicalimaging technique used to generate the medical imaging data. Forexample, for X-ray imaging, the value of an element (e.g., pixel) maycorrespond to or represent a degree to which X-rays have been detectedat the particular part of the imaging plane corresponding to theelement. As another example, for Magnetic Resonance Imaging, the valueof an element (e.g., voxel) may correspond to or represent a rate atwhich excited nuclei, in a region corresponding to the element, returnto an equilibrium state. In some examples, each element may only haveone value. However, in other examples, each element may have orotherwise be associated with multiple values. For example, the multiplevalues of a given element may represent the values of respectivemultiple signal channels. For example, each signal channel may representa different medical imaging signal or property of the imaging subject.In some examples, the at least one value may comprise an element (e.g.,pixel or voxel) intensity value. For example, an output signal from themedical imaging may be mapped onto a pixel or voxel intensity value, forexample a value within a defined range of intensity values. For example,for a greyscale image, the intensity value may correspond to a value inthe range 0 to 255, where 0 represents a ‘black’ pixel and 255represents a ‘white’ pixel, for example. As another example, for exampleas in the case of USHORT medical imaging data, the intensity value maycorrespond to a value in the range 0 to 65536. As another example, in acolor image (e.g., where different colors represent different propertiesof the imaging subject) each pixel/voxel may have three intensityvalues, e.g., one each for Red, Green, and Blue channels. It will beappreciated that other values may be used. In any case, the medicalimaging data may be rendered into an image, for example as schematicallyillustrated in FIGS. 2 to 5 . In the illustrated examples, the firstmedical imaging data 220 and the second medical imaging data 330 aredata captured by performing medical imaging on the same patient,specifically the same region of the same patient, but at differenttimes. In some examples, the modality (i.e., the medical imaging methodby which the data was captured) and/or the protocol (i.e., the specificparameters by which a given method of medical imaging was performed) ofthe medical imaging may alternatively or additionally be differentbetween the first medical imaging data 220 and the second medicalimaging data 330.

In any case, as can be seen from FIGS. 2 to 5 , the first medicalimaging data 220 and the second medical imaging data 330 differ.Specifically, in this example, the data differ in the positioning of theregion in which the imaging data was captured relative to thepatient—the region is offset to the right in the sense of the Figures inthe second medical imaging data 330 of FIGS. 3 to 5 as compared to thefirst medical imaging data 220 in FIG. 2 . Nonetheless, certain features226, 242 are represented in both the first medical imaging data 220 andthe second medical imaging data 330. In this example, a given feature226 represented in the medical imaging data is a lesion 226 in the rightlung of the patient. It will be appreciated however that in examples thegiven feature may be any feature, e.g., any particular part of theimaging subject (e.g., including internal cavities and the like),represented in the medical imaging data.

In an example use case (referred to hereinafter for illustrativepurposes), a physician may be reviewing the rendering of the firstmedical imaging data 330 (as per that illustrated in FIG. 2 ). Thephysician may be interested in assessing the progression of a givenfeature, e.g., the lesion 226, since the previous, second medicalimaging data 330 was captured. The location 224 at which the givenfeature 226 is represented in the first medical imaging data 220 isknown. However, the location at which the given feature 226 isrepresented in the second medical imaging data 330 is not known and maybe difficult or burdensome for the physician to ascertain by visualinspection alone. The method illustrated in FIG. 1 determines thelocation at which the given feature 226 is represented in the secondmedical imaging data 330.

Referring again to FIG. 1 , in broad overview, the method comprises:

-   -   in step 102, obtaining a first descriptor for a first location        224 in first medical imaging data 220, the first location 224        being the location within the first medical imaging data 220 at        which the given feature 226 is represented, the first descriptor        being representative of values of elements 222 of the first        medical imaging data 220 located relative to the first location        224 according to a first predefined pattern;    -   in step 104, obtaining a second descriptor for each of a        plurality of candidate second locations 334, 340, 440, 448 in        second medical imaging data 330, each second descriptor being        representative of values of elements 332, 338 of the second        medical imaging data 330 located relative to the respective        candidate second location 334, 340, 440, 448 according to the        first predefined pattern;    -   in step 106, calculating, for each of the plurality of candidate        second locations 334, 340, 440, 448 a similarity metric        indicating a degree of similarity between the first descriptor        and the second descriptor for the candidate second location 334,        340, 440, 448;    -   in step 108, selecting a candidate second location 334, 448 from        among the plurality of candidate second locations 334, 340, 440,        448 based on the calculated similarity metrics; and    -   in step 110, determining the location 334, 446 at which the        given feature 226 is represented in the second medical imaging        data 330 based on the selected candidate second location 334,        448.

Accordingly, a technique for determining the location 334, 446 at whicha given feature 226 is represented in medical imaging data 330 isprovided for. Specifically, a known location at which a given feature226 is represented in a first medical image 220 is used to determine thelocation at which the given feature is represented in a second, e.g.,previous, medical image 330 of a patient. This may, for example, reducethe burden for a physician in finding the location at which the givenfeature 226 is represented in the second medical image 330.

Moreover, this determination is based on determining the similaritybetween descriptors for the known location at which the feature 226 isrepresented in the first medical image 220 and for each of plurality ofcandidate locations in the second medical image 330. This may providefor fast, efficient, and/or flexible feature location.

For example, determining similarity between the descriptors for theknown first location 224 and the candidate second locations 334, 340 maybe significantly less computationally expensive as compared to ImageRegistration techniques where every pixel/voxel in one image is mappedto every pixel/voxel in another image by optimizing a cost function.Accordingly, for a given computational budget, the presently disclosedmethod may provide results significantly faster than Image Registrationbased methods. This may, in turn, allow for real-time or near real-timeinteraction with the image data.

Basing the descriptor on element (e.g., pixel of voxel) values locatedrelative to the given location 224 in a predetermined distributionpattern allows for the surroundings and spatial context of the feature226 to be encoded into the descriptor. This provides for the location atwhich the given feature 226 is represented in the second medical imagingdata 330 to be determined in a reliable, efficient and/or flexiblemanner.

For example, such descriptors may encode the surroundings of the featureand candidate locations of interest, rather than attempting to map everypixel of one image to a pixel of another image as per Image Registrationtechniques. Accordingly, even where the first 220 and second 330 imagesare relatively different (e.g., in the overall region of the body theydepict), the location at which the given feature is represented in thesecond medical image may nonetheless be reliably determined (e.g., ascompared to Image Registration techniques which, due to the attempt tomap every pixel between images, are typically limited to images that arerelatively similar). The presently disclosed technique may thereforeprovide accurate results for a broader range of first and second images,and hence may be more flexibly applied.

As another example, determining similarity between the descriptors forthe known first location 224 and the candidate second locations 334, 340need not rely on the presence in the medical images 220, 330 of‘landmarks’ that classifiers have been trained to detect in the images,as per Landmark Detection based methods. According, the presentlydisclosed method may be more flexible with respect to the types ofmedical images to which it may be effectively applied. Moreover, bybasing the location determination on a similarity between descriptors,the presently disclosed technique can be applied for any given feature,rather than e.g., a landmark on which a classifier has been trained asper landmark detection-based methods. The presently disclosed techniquemay therefore provide accurate results for a broader range features, andhence may be more flexibly applied. Determining the similarity betweenthe descriptors allows for the location at which a given feature isrepresented in medical imaging data to be determined without the use oftrained classifiers as per landmark detection based techniques, andhence the time and effort associated with preparing a training data setfor the classifier, as well as the computational load of training theclassifier, can be saved. Accordingly, the presently disclosed methodmay allow for determination of a location at which a given feature 226is represented in medical imaging data in an efficient manner.

As mentioned, the method comprises, in step 102, obtaining the firstdescriptor for the first location 224 in first medical imaging data 220.The first location 224 is the location within the first medical imagingdata 220 at which the given feature 226 is represented. The firstdescriptor is representative of values of elements 222 of the firstmedical imaging data 220 located relative to the first location 224according to a first predefined pattern.

In some examples, the first descriptor may be output from a descriptormodel applied to the first medical imaging data for the first location224. The descriptor model may be configured to calculate a descriptorfor a given location 224 based on the values of elements locatedrelative to the given location 224 according to the first predefinedpattern.

In some examples, the first descriptor may be obtained from a database(not shown). For example, the descriptor for the first location 224 mayhave already been calculated (for example by applying the descriptormodel), and stored in the database, for example in association with thefirst location 224. For example, the database may store a plurality offirst descriptors each in association with the corresponding firstlocation in the medical imaging data on the basis of which the firstdescriptor was determined. Accordingly, in some examples, the method maycomprise selecting the first location 224 from among the plurality andextracting the first descriptor associated with the selected firstlocation 224.

In either case, a descriptor for a given location 224 may be a vectorcomprising a plurality of entries, each entry being representative ofvalues of a set of one or more elements, the sets of one or moreelements being located relative to the given location 224 according tothe first predefined pattern. For example, each entry may berepresentative of the values of the elements located within a respectiveone or more of a plurality 222 of predefined boxes 223 (i.e.,rectangular regions) located relative to the given location 224according to the first predefined pattern. It will be appreciated that,where the medical imaging data exits in three spatial dimensions, theterm ‘box’ as used herein may refer to a cuboidal region or volume.

In some examples, each entry of the descriptor may be representative ofthe values of the elements located within a respective one of aplurality 222 of predefined boxes 223. For example, each entry of thedescriptor may be an average of the values of the elements locatedwithin a respective one of a plurality 222 of predefined boxes 223. Thatis, each entry may be the sum of the values of the elements locatedwithin a particular box 223, divided by the number of elements includedin the box 223. For example, as illustrated in FIG. 2 , for the firstlocation 224, there are a plurality 222 of predefined boxes 223 (i.e.,notional regions) distributed in the first medical imaging data 220 in aparticular pattern 222. The first descriptor for the first location 224may be a vector, each entry of which is the average value of theelements of the first medical imaging data 220 located within arespective one of the boxes 223. Using the average value (e.g., ascompared to the sum) helps provide that each vector entry is within thesame range, independent of the size of the box for which it iscalculated. As described in more detail below, this may, in turn, helpprovide for robust and/or reliable determination of similarity betweendescriptors.

In some examples, the predefined pattern and/or the predefined boxes(e.g., the size and/or aspect ratio of each box) may be randomly orpseudo-randomly generated. In some examples, a descriptor may bedetermined using many boxes 223, for example 1000 boxes, and accordinglythe descriptor may be a vector having many entries (e.g., 1000 entries).For example, referring briefly to FIG. 6 , there is presented, forillustrative purposes, a medical imaging data set 660 to which a largenumber of predefined boxes (shown as white rectangular outlines) havebeen applied in order to determine a descriptor for a given location(not shown) in the medical imaging data 660.

The descriptor may encode the spatial context of the given location 224at which a given feature is represented, and hence in turn may provide acompact representation of the surroundings of a given feature. Thecalculation of such descriptors may be relatively computationallyinexpensive and fast, for example as compared to comparatively densefeature representations, for example as may be used in a landmarkdetection technique. This may help allow, for example, for the method tobe performed (and hence results returned) quickly.

In some examples, the descriptor model that calculates the descriptormay be applied to ‘raw’ medical imaging data. However, in otherexamples, the descriptor model may be applied to integral image data(also known as a summed area table) of the first medical imaging data.In integral image data, the value for a given element is the sum ofvalues of all of the elements above and to the left of the given elementin the image data. For example, integral image data for the firstmedical imaging data 220 may be generated and the first descriptor maybe calculated on the basis of the integral image data for the firstmedical imaging data 220. The use of integral image data allows forfaster computation of the descriptors. In some examples, this may, inturn, help allow for the results of the method to be returned faster.

In examples where an integral image is used, the sum of values ofelements of a box 223 with opposite corner locations (x₁,y₁,z₁),(x₂,y₂,z₂) is given, in terms of the corresponding Integral image I, by(I(x₂,y₂,z₂)+I(x₂,y₁,z₁)+I(x₁,y₂,z₁)+I(x₁,y₁,z₂)−(I(x₁,y₂,z₂)+I(x₂,y₁,z₂)+I(x₂,y₂,z₁)+I(x₁,y₁,z₁))).In this expression, I(x_(i),y_(j),z_(k)) is the value of the element inthe integral image I at the location x=i, y=j, and z=k, where i, j, andk are element indices. For a given box 223, this sum may be divided bythe total number of elements contained within the box 223 to calculatethe average element value for the box 223. The average element value foreach box may be used as respective entry in a vector constituting thefirst descriptor.

It will be appreciated that, in some examples, descriptors other thanthe specific example described above may be used. For example, in someexamples, Haar-like descriptors may be used, i.e., a descriptor whereeach entry represents a difference between the sums of element valueswithin each of a plurality of boxes defined in the image data. In someexamples, the descriptor may be a gradient descriptor, for example inwhich each entry represents one or more image gradients in a respectiveone of a plurality of regions of the medical imaging data. For example,an image gradient for a given region may be based on a change in thevalues (e.g., intensity values) between elements within the givenregion. In some examples, the descriptor may be such that each entry isthe value of a respective one of a plurality of elements randomlydistributed in the medical imaging data relative to the first location224. In some examples, the descriptor for a given location may be suchthat each entry is the aggregate of the values of elements intersectingwith a respective one of a plurality of randomly orientated rays, eachray originating from the given location. In each case, the descriptorfor a given location is representative of values of elements of themedical imaging data located relative to the given location according toa first predefined pattern.

Nonetheless, is noted that the inventors have identified that the use ofa descriptor for a given location in which each entry is representative(e.g., an average) of the values of the elements located within arespective one of a plurality 222 of predefined boxes 223 locatedrelative to the given location in a predefined (e.g., randomlygenerated) distribution pattern, provides for particularly fast yetaccurate location determination.

In some examples, the first location 224 for which the descriptor iscalculated or otherwise obtained may be specified by a user. Forexample, a representation of the first medical imaging data 220 may bedisplayed to the user, and the user may specify the location 224 of agiven feature 226 of interest, for example by clicking on therepresentation at the first location 224 at which the given feature 226is represented. This user specified location may then be taken as thefirst location 224. The first descriptor may then be calculated orotherwise obtained based on this first location 224.

In some examples, the first location 224 may be output from a computerimplemented method. For example, the first medical imaging data 220 mayhave been pre-processed by a computer implemented method (not shown) toidentify a given feature 226 in the first medical imaging data 220, andoutput the first location 224 at which the given feature 226 isrepresented. This output may be provided directly, and/or stored in adatabase. The first descriptor may then be calculated or otherwiseobtained based on this first location 224.

In some examples, the first location 224 may be obtained from a database(not shown). For example, the database may store one or more locationsat which a respective one or more features are represented in one ormore medical imaging data sets. The first medical imaging data 220 maybe extracted from the database along with the one or more locations. Aparticular one of the locations may be selected as the first location224, for example based on a desire or instruction to determine thelocation at which the given feature 226 at that location is representedin second medical imaging data 330. The first descriptor may then becalculated or otherwise obtained based on this first location 224.

In any case, the first descriptor for the first location 224 in thefirst medical imaging data 220 is obtained.

As mentioned, in step 104, the method comprises obtaining a seconddescriptor for each of a plurality of candidate second locations 334,340 in second medical imaging data 330.

Each second descriptor is representative of values of elements 332, 338of the second medical imaging data 330 located relative to therespective candidate second location 334, 340 according to the firstpredefined pattern. The second descriptor may be the same as the firstdescriptor in the sense that, for a given location (e.g., the firstlocation 224 or any one of the second candidate locations 334, 340), thedescriptor is representative of values of given elements of theassociated medical imaging data located relative to the given locationaccording to the first predefined pattern. For example, the samedescriptor model that was applied to the first medical imaging data 220to generate the first descriptor for the first location 224 may beapplied to the second medical imaging data 330 to generate the seconddescriptor for each of the plurality of candidate second locations 334,340. For example, referring to FIGS. 2 and 3 , the boxes 332, 338 andthe location of each of those boxes relative to each candidate secondlocation 334, 340 used to calculate the second descriptor for each ofthose candidate second locations 334, 340 are the same as the boxes 222and the location of each of those boxes relative to the first location224 used to calculate the first descriptor for the first location 224.Whichever type of descriptor is used, that descriptor for the firstlocation 224 in the first medical imaging data 220 and that descriptoreach of the plurality of second candidate locations 334, 340 in thesecond medical imaging data 330 are obtained.

As described in more detail below, in some examples (e.g., describedbelow with reference to FIG. 3 ), each candidate second location 334,340 may be a location at which a respective previously detected feature226, 242 is represented in the second medical imaging data 330. In otherexamples, (e.g., as described below with reference to FIG. 4 ) thecandidate second locations 440 may be locations distributed through thesecond medical imaging data in a second predefined pattern.

In any case, a second descriptor for each of a plurality of candidatesecond locations 334, 340, 440 in second medical imaging data 330 isobtained.

As mentioned, the method comprises, in step 106 calculating, for each ofthe plurality of candidate second locations 334, 340, 440, a similaritymetric indicating a degree of similarity between the first descriptorand the second descriptor for the candidate second location 334, 340,440.

In some examples, the similarity metric may comprise the normalizedmutual information similarity between the first descriptor and thesecond descriptor. For example, the normalized mutual informationsimilarity between the first descriptor and the second descriptor may bedetermined as follows. A first histogram is formed in which the entriesin the first descriptor are placed into equally sized bins x between theminimum entry value and the maximum entry value of the first descriptor.The counts in first histogram are normalized to get the probabilitydistribution Px(x) of the entries of the first descriptor across thebins x. A second histogram is formed in which the entries in the seconddescriptor are placed into equally sized bins y between the minimumentry value and the maximum entry value of the second descriptor. Thecounts in second histogram are normalized to get the probabilitydistribution Py(y) of the entries of the second descriptor across thebins y. A joint histogram of the entries of the first descriptor and thesecond descriptor ranging between the respective minimum and maximumvalues is determined. Each bin of the joint histogram is an equallysized 2-dimensional bin, the first dimension X corresponding to theentry from the first descriptor, and the second dimension Ycorresponding to the associated entry from the second descriptor. Forexample, if the first entry of the first descriptor was q and the firstentry of the second descriptor was p, then the 2D bin x, y of the jointhistogram which covers a range of first descriptor values including pand covers a range of second descriptor values including q, wouldreceive a count. The counts in the joint histogram are normalized to getthe probability distribution Pxy(x,y) of the entries of the first andsecond descriptors across the bins x, y. The mutual informationsimilarity I between the first descriptor and the second descriptor maythen be calculated as

$\begin{matrix}{I = {\sum_{y}{\sum_{x}{{P_{XY}\left( {x,y} \right)}\log\left( \frac{P_{X,Y}\left( {x,y} \right)}{{P_{X}(x)}{P_{Y}(y)}} \right)}}}} & (1)\end{matrix}$

The higher the mutual information similarity I, the higher the degree ofsimilarity between the first descriptor and the second descriptor. Usingnormalized mutual information similarity may provide for robust,reliable and/or flexible determination of the similarity. For example,the mutual information similarity is independent of differences inrelative scale of the entries of the first descriptor as compared to thesecond descriptor. For example, using mutual information, an accuratesimilarity metric may be determined even if the overall ‘brightness’(e.g., the intensities that are the values of the elements of themedical imaging data) differs between the first medical imaging data andthe second medical imaging data. As another example, mutual informationmay provide an accurate similarity, even in cases where differentprotocols and/or modalities of medical imaging have been used (and e.g.,accordingly different value ranges used or achieved) or for examplewhere the medical images have been transformed, e.g., inverted orscaled. Accordingly, the use of mutual information similarity in thisway may provide for determination of similarity that is robust to, e.g.,non-structural variations between the first and second medical imaginedata, which may in turn allow for the method to be applied reliably to awider range of images, which in turn may provide more flexibility in thetype of images with which the method may be provided.

As mentioned, in some examples, the descriptor entries arerepresentative of values of elements contained within associated boxes223, 333, 339. As mentioned, in these examples, each entry of therespective descriptors being an average of the values of elementscontained within respective boxes 223, 333, 339 may help ensure theentries within a descriptor are within a certain range independent ofbox size. This, in turn, may facilitate the use of mutual informationsimilarity, as the bin into which a given entry is placed is accordinglydependent on the average value of the elements within the box, but noton the size of the box.

In some examples, other similarity metrics between the first descriptorand the second descriptor may be used. For example, cosine similarity,Euclidean distance, and/or cross correlation may alternatively oradditionally be used. Nonetheless, the inventors have identified thatthe mutual information similarity metric may provide for particularlyrobust, reliable and/or flexible determination of the similarity metric,and accordingly for particularly robust, reliable and/or flexibledetermination of the location at which the given feature is representedin the second medical imaging data.

In any case, for each second candidate location 334, 340 a similaritymetric indicating a degree of similarity between the first descriptorand the second descriptor for the candidate second location 334, 340 iscalculated.

As mentioned, the method comprises, in step 108, selecting a candidatesecond location 334 from among the plurality of candidate secondlocations 334, 340 based on the calculated similarity metrics; and instep 110 determining the location 334 at which the given feature 226 isrepresented in the second medical imaging data 330 based on the selectedcandidate second location 334.

In some examples, selecting the candidate second location 334 maycomprise selecting the candidate second location 334 having thesimilarity metric indicating the highest degree of similarity among thesimilarity metrics of the plurality of candidate second locations 334,340. For example, the candidate second location 334 with the highestmutual information similarity metric may be selected. In some examples,determining the location at which the given feature 226 is representedcomprises determining, as the location at which the given feature 226 isrepresented in the second medical imaging data 330, the selectedcandidate second location 334. In some examples, determining thelocation may be responsive to a determination that the similarity metricbetween the first descriptor for the first location and the seconddescriptor for the selected candidate second is above a threshold value.This may help ensure that the location at which the given feature isrepresented in the second medical imaging data 330 is only determined incases where there is a certain degree of confidence in thedetermination. This may, in turn, help provide for reliabledetermination of the location at which the given feature is representedin the second medical imaging data 330.

As mentioned, in some examples (e.g., as illustrated in FIG. 3 ), eachcandidate second location 334, 340 may be a location at which arespective previously detected feature 226, 242 is represented in thesecond medical imaging data 330. In other examples, (e.g., asillustrated in FIG. 4 ) the candidate second locations 440 may belocations distributed through the second medical imaging data in asecond predefined pattern. The way in which the location of the givenfeature may be determined in each of these examples scenarios is nowdescribed in more detail with reference to FIGS. 3 and 4 .

Referring first to FIG. 3 , in this example scenario, the second medicalimaging data 330 represents a first feature 226 and a second feature242. In this example, these features 226, 242 have been detected in thesecond medical imaging data 330, for example previously by a physician,or by an automated method. The respective locations 334, 340 of thesefeatures 226, 242 in the second medical imaging data 330 have beenrecorded as part of this previous detection. Accordingly, the respectivelocations 334, 340 of these previously detected features 226, 340 areknown. However, it is not known which of these two locations 334, 340 isthe location at which the given feature 226 in the first medical imagingdata 220 is represented in the second medical imaging data 330. In otherwords, it is not known which of these features 226, 224 corresponds tothe given feature 226 represented in the first medical imaging data 220.In these examples, the candidate second locations may therefore be takenas the locations 334, 340 in the second medical imaging data 330 atwhich the previously detected features 226, 242 are represented. Asimilarity metric between the second descriptors for each of thecandidate second locations 334, 340 and the first descriptor for thefirst location 224 may be calculated. In this example, the candidatesecond location 334 with the highest similarity metric may be selected,and the selected candidate second location 334 may be determined as thelocation 334 at which the given feature 226 is represented in the secondmedical imaging data 330.

Referring to FIG. 4 , the second medical imaging data 330 represents afirst feature 226 and a second feature 242. However, in the examplescenario of FIG. 4 , the locations at which these features 226, 242 arerepresented in the second medical imaging data 330 is not known. In thisexample, the candidate second locations 440 are locations distributedthrough the second medical imaging data 330 in a second predefinedpattern. In some examples, as illustrated in FIG. 4 , the secondpredefined pattern may be a regular arrangement, with each secondcandidate location 440 (represented in FIG. 4 by the open circles 440)spaced apart from its neighbors in a grid. For example, this arrangementmay span a region, for example the whole of, the second medical imagingdata 330. As such, the candidate second locations represent a spatiallydistributed sample of locations within the second medical imaging data220 (and hence at which the given feature could be represented).

The candidate second locations being distributed in the secondpredefined pattern may allow for the location at which the given feature226 is represented in the second medical imaging data 330 to beestimated in a computationally efficient way, for example as compared todetermining a descriptor for every voxel of the second medical imagingdata 330.

In the example of FIG. 4 , the similarity metric between the seconddescriptor for each one of the candidate second locations 440 and thefirst descriptor may be calculated. As illustrated in FIG. 4 , thesecond descriptor for the candidate second location 448 closest to therepresented feature 226 has the highest degree of similarity with thefirst descriptor. Accordingly, the candidate second location 448 isselected. The location at which the given feature 226 is represented inthe second medical imaging data 330 may then be determined based on thisselected candidate second location 448.

For example, in some examples, the selected candidate second location448 may be taken as an estimate of the location at which the givenfeature 226 is represented in the second medical imaging data. However,in other examples, this estimate may be refined by defining furthercandidate locations 442, 444 (based on the selected candidate secondlocation 448) in successively more fine-grained patterns (see e.g., thepattern of grey circles 442, and subsequently of black circles 444, inFIG. 4 ). Accordingly, an accurate yet efficient determination of thelocation at which a given feature 226 is represented in the secondmedical imaging data 330 ma be provided for, even where no locations ofany features in the second medical imaging data 330 are known.

Specifically, in these examples, determining the location 446 at whichthe given feature 226 is represented in the second medical imaging data330 may comprise: determining, based on the selected candidate secondlocation 448, a plurality of candidate third locations 442 in the secondmedical imaging data 330. For example, the candidate third locations 442may be defined as locations in the region of (i.e., local to) theselected candidate second location 448. For example, the candidate thirdlocations 442 may be locations distributed through the second medicalimage data 330 in a third predefined pattern in the region of theselected second location 448. For example, the third predefined patternmay be the same as or similar to that of the second predefined pattern.However, in some examples (as illustrated in FIG. 4 ), a distancebetween the candidate third locations 442 in the third predefinedpattern may be less than a distance between the candidate secondlocations 440 in the second predefined pattern.

The method may then comprise obtaining a third descriptor for each ofthe plurality of candidate third locations 442 in the second medicalimaging data 330, each third descriptor being representative of valuesof elements of the second medical imaging data 330 located relative tothe respective candidate third location 442 according to the firstpredefined pattern. For example, the third descriptor may be the sameas, i.e., may have been calculated in the same way as, the first andsecond descriptors as described above.

In some examples, a scale of the first predefined pattern associatedwith the third descriptors may be reduced as compared to a scale of thefirst predefined pattern associated with the second descriptors. Takinga descriptor based on a plurality of boxes (not shown in FIG. 4 )arranged in the first predefined pattern as an example, while the shapeand the spatial pattern of the boxes may be the same for the seconddescriptors and the third descriptors, the size of the boxes and/or thedistances between each box and the respective location 442 for the thirddescriptor may be scaled down (i.e., reduced) as compared to the size ofthe boxes and/or the distances between each box and the respectivelocation 440 for the second descriptor. This may provide that the thirddescriptors encode more fine-grained detail of the second medical imagedata 330 in the region of the selected candidate second location 448.This may help provide for accurate determination of the location atwhich the given feature 226 is represented in the second medical imagingdata 330.

In these examples, the method may then comprise calculating, for each ofthe plurality of candidate third locations 442, a similarity metricindicating a degree of similarity between the first descriptor and thethird descriptor for the candidate third location 442. For example, thesimilarity metric may be the same as, i.e., may be calculated in thesame way as, the similarity metric as used between the first descriptorand the second descriptor. The method may then comprise selecting acandidate third location 448 from among the plurality of candidate thirdlocations 442 based on the calculated similarity metrics indicating thedegree of similarity between the first descriptor and the respectivethird descriptors. For example, as illustrated in FIG. 4 , the candidatethird location 448 with the highest similarity metric may be selected.The method may then comprise determining the location at which the givenfeature 226 is represented in the second medical imaging data 330 basedon the selected candidate third location 448.

In some examples, the selected candidate third location 448 may be takenas an estimate of the location at which the given feature 226 isrepresented in the second medical imaging data. However, in otherexamples, this estimate may be further refined by defining furthercandidate locations 444 based on (e.g., in the region of) the selectedthird candidate location 448, and repeating the above described methodfor these further candidate locations. For example, as illustrated inFIG. 4 , a further set of candidate locations 444 are defined in theregion of the selected candidate third location 448. In this example thefurther candidate location 446 will have a descriptor with the highestsimilarity metric with the first descriptor, and hence may be selected.The location at which the given feature 226 is represented in the secondmedical imaging data 330 may then be determined at the selected furthercandidate location 446. It will be appreciated that although threelevels of hierarchical granularity are shown in FIG. 4 , fewer or morelevels of hierarchical granularity may be employed in order to determinethe location at which the given feature 226 is represented in the secondmedical imaging data 330. Nonetheless, using such a hierarchical methodmay allow for the location to be accurately determined in a quick andefficient manner, even where no locations of any features in the secondmedical imaging data 330 are known.

It is noted that the elements (e.g., boxes) according to which therespective second, third or further descriptors may calculated for eachof the respective candidate second, third or further locations are notshown in FIG. 4 , for clarity.

In any case, the location at which the given feature 226 is representedin the second medical imaging data 330 is determined. In some examples,the method may further comprise generating output data indicating thedetermined location at which the given feature 226 is represented in thesecond medical imaging data 330.

For example, the output data may comprise a coordinate or pixel/voxelindex corresponding to the determined location 334, 446 within thesecond medical imaging data 330. In some examples, the output data mayfurther comprise a reference to the second medical imaging data 330within which location has been determined. In some examples, the outputdata may comprise the second medical imaging data itself (or a portionthereof). In some examples, the output data may comprise an image (ordata for an image) in which an indicator indicating the determinedlocation is overlaid onto a rendering of the second medical imaging data330. For example, as illustrated in FIG. 5 , a representation of thesecond medical imaging data 330 is shown, and overlaid onto (orotherwise included in) the representation is an indicator 550 indicatingthe location at which the given feature 226 is represented in the secondmedical imaging data 330. In this example, the indicator is a boxcentered on the determined location. However in other examples otherindicators may be used, such as a marker or dot or other symbol overlaid(or otherwise included) at the determined location; or for example anarrow or pointer or other label pointing at or connected or otherwiseindicating the determined location. The output data may allow for anindication to be provided to a user, e.g., a physician, as to thedetermined location at which the given feature 226 is represented in thesecond medical imaging data 330. The user may therefore, e.g., comparethe representation of the given feature 226 in both the first medicalimage data 220 and the second medical image data 330. As a result, theuser may, for example, make an assessment as to the differences betweenthe given feature 226, e.g., make an assessment as to the progression ofa disease of which the feature 226 may be an expression.

In some examples, the output data may further comprise the firstlocation and a reference to the first medical imaging data 220 or themedical imaging data 220 itself. This may allow for an associationbetween the locations at which the given feature 226 is represented inboth the first medical imaging data 220 and the second medical imagingdata 330 to be determined. In some examples, the output data may bestored in a storage device. For example, the output data may be storedin a database. This may allow for the output data to be referred to, forexample by a user or by an automated downstream process (not shown).

In some examples, the method may comprise transmitting the output datato a display device to display a representation of the second medicalimage data 330 and an indicator 550 indicating, on the representation ofthe second medical image data, the determined location at which thegiven feature is represented. For example, the display device (notshown) may be a computer monitor or other display screen of a computer.For example, the displayed representation may be similar to that shownin FIG. 5 , where the indicator 550 is overlaid onto the representationof the second medical imaging data 330. Any of the example indicatorsmentioned above may be used. This may allow for a user, e.g., aphysician, to immediately and easily appreciate the location at whichthe given feature 226 is represented in the second medical imaging data330, which may, for example, allow for assessments based on the givenfeature to be made more quickly and with minimal burden.

In the above examples, only one given feature 226 is referred to.However, in some examples, there may a plurality of given featuresrepresented in the first medical imaging data 220. In these examples, itmay be desirable to determine the location at which each of theplurality of given features is located in the second medical imagingdata 330. In some examples, the method according to any of the examplesdescribed with reference to FIGS. 1 to 6 may be employed sequentially tothe plurality of given features, for example once for each of theplurality of given features. However, as described in more detail below,in some examples, the determination of each location may be performedconcurrently.

Referring to FIG. 7 , there is a representation of first medical imagingdata 772, and a representation of second medical imaging data 770. Thefirst medical imaging data 772 represents three given features A, B, C(shown only schematically in FIG. 7 ), and the second medical imagingdata 770 represents three features D, E, F (shown only schematically inFIG. 7 ). In this example, all of the features A, B, C, D, E, F havebeen detected in the images, and the locations at which they arerepresented within the respective medical imaging data is known.However, it is not known which of the features D, E, F in the secondmedical imaging data 770 correspond to the given features A, B, C in thefirst medical imaging data. In this example, the locations at which thegiven features A, B, C are represented in the first medical imaging data772 are the first locations, and the locations at which the features D,E, F are represented in the second medical imaging data 770 are therespective candidate second locations.

In this example, a said first descriptor for a said first location (notshown) in the first medical imaging data 770 is obtained for each of theplurality of said given features A, B, C. In this example, the methodmay comprise calculating said similarity metric between the firstdescriptor and the second descriptor for all pairs of the firstlocations A, B, C and candidate second locations D, E, F. For example,the similarity metric according to any one of the examples describedabove may be determined between the descriptor for the first location ofthe given feature A and the descriptor for the second locations of eachof the features D, E, F; and the same may be performed for the othergiven features B, C. For example, in this example, there will be ninesimilarity metrics calculated, one for each of the pairings A-D, A-E,A-F, B-D, B-E, B-F, C-D, C-E, C-F.

The method may then comprise assigning each first location A, B, C adifferent one of the candidate second locations D, E, F based on thecalculated similarity metrics between the pairs of first locations andcandidate second locations. For example, the assignment may be bysolving the maximal matching problem for a bipartite graph based on thesimilarity metrics of all of the pairs. For example, the assignments maybe made using a Linear Sum Assignment algorithm, for example the “linearsum assignment” routine in a Python-NumPy library. For example, theassignment may be made by minimizing the function Σ_(i)Σ_(j)C_(ij)X_(ij)where C is a cost matrix where C[i,j] is the ‘cost’ of assigning firstlocation i a candidate second location j (e.g., inversely proportionalto the similarity metric thereof), X is a Boolean matrix where X[i,j]=1if row i is assigned to column j, and wherein the minimization issubject to the constraint that there is at most one candidate secondlocation j assigned per first location i. For example, as a result ofthis process, the following assignments may be determined: A-D, B-E,C-F.

In some examples, assigning a first location A, B, C a candidate secondlocation D, E, F may be responsive to a determination that thesimilarity metric between the first descriptor for the first location A,B, C and the second descriptor for the candidate second location D, E, Fis above a threshold value. For example, the threshold value could beset, for example, at 70% similarity. This may help ensure that there isa certain degree of confidence in the assignment. For example, whilegiven feature A may have been assigned to feature D in the secondmedical imaging data 770 as part of the linear sum assignment, it may bedetermined that the similarity metric for this pair A-D is below thethreshold value, and hence the assignment of D to A may not be made.However, the similarity metrics for the pairings B-E and C-F may beabove the threshold, and hence these assignments may be made.

The method may then comprise determining, for each said first locationB, C, as the location at which the respective said given feature B, C isrepresented in the second medical imaging data 770, the second candidatelocation E, F assigned to the first location. For example, according tothe assignments B-E and C-F made in this example, the method maycomprise determining the locations of features E and F as the locationat which features B and C, respectively, are represented in the secondmedical imaging data 770.

The method described above with reference to FIG. 7 may help preventinconsistencies that could otherwise occur if the method described abovewith reference to FIGS. 1 to 6 were applied sequentially and separatelyto each of a plurality of given features. Accordingly, this may allowfor more reliable determination of the location at which a given featureis represented in the second medical imaging data 770 in the case wherethere are a plurality of given features.

In some examples, the output data may comprise associations or links774, 776 between each first location B, C and the candidate secondlocation E, F that has been assigned to the first location B, C. Forexample, the output data may be similar that represented in FIG. 7 ,where representations of the first medical imaging data 772 and thesecond medical imaging data 770 are presented adjacent to one another,and where there is a first link 774 drawn between the first location Band the candidate second location E to which it has been assigned, and asecond link 776 drawn between the first location C and the candidatesecond location F to which it has been assigned. These links 774, 776may act as indicators of the locations E, F at which the given featuresB, C are represented in the second medical imaging data 770. This outputdata may be stored and/or displayed, for example as described above.This may allow a user to immediately and easily identify the location atwhich each given feature B, C represented in the first medical imagingdata 772 is represented in the second medical imaging data 770, andindeed vice versa.

To serve as an illustration of the performance of examples of the methoddisclosed herein, a study was performed. Specifically, a study wasperformed for examples described above with reference to FIG. 4 where ahierarchical search for the location at which the given feature 226 isrepresented in the second medical imaging data 330 is performed. In thisstudy, the method was performed on a benchmark dataset of lung lesionpairs each across two images taken at different time points (i.e., firstmedical imaging data and second medical imaging data, respectively). Inthis study, the distance from the estimated location at which the givenfeature is represented in the second medical imaging data to the truelocation at which the given feature is represented in the second medicalimaging data, was calculated, for each of the dataset pairs. Theprecision of the estimation under a series of distance thresholds wasthen calculated. This provides a measure of the sensitivity of theestimation as a function of the distance from the true location at whichthe given feature is represented in the second medical imaging data.FIG. 8 illustrates a plot of this sensitivity as a function of thedistance (in mm) for the above referenced example of the presentlydisclosed method (solid line 882) as well as comparative methods (dashedline 884, dotted line 880, and black circles 886) as applied to thedataset. The comparative methods were each Image Registration basedmethods. As can be seen from the plot in FIG. 8 , the presentlydisclosed method can surpass the performance of the studied ImageRegistration based methods. Specifically, in about 92% of the cases, thestudied example method was able to determine the location at which thegiven feature was represented in the second medical imaging data 330 towithin 15 mm of its true location. In the studied example method, thelocation was not only able to be determined accurately, but alsoquickly, specifically in under a few seconds. According, the presentlydisclosed method can provide for fast and accurate determination of thelocation at which the given feature is represented in the second medicalimaging data 330.

Referring to FIG. 9 , there is illustrated an apparatus 990 according toan example. The apparatus 990 comprises an input interface 996, anoutput interface 998, a processor 992, and a memory device 994. Theprocessor 992 and the memory device 994 may be configured to perform themethod according to any one of the examples described above withreference to FIGS. 1 to 7 . The memory device 994 may store instructionswhich, when executed by the processor 992, cause the processor 992 toperform the operations of the method according to any one of theexamples described above with reference to FIGS. 1 to 7 . Theinstructions may be stored on any computer readable medium, for example,one or more non-transitory computer readable media.

For example, the input interface 996 may receive the first descriptorfor a first location 224, A, B, C in first medical imaging data 220, 772and a second descriptor for each of a plurality of candidate secondlocations 334, 340, 440, D, E, F in second medical imaging data 330,770, the processor 992 may implement the method according to any of theexamples described above with reference to FIGS. 1 to 7 , and theprocessor 992 may output, via the output interface 998, data indicatingthe determined location at which the given feature is represented in thesecond medical imaging data 330, 330, for example the output data asdescribed above with reference to FIG. 5 or 7 . In some examples, theoutput data may be transmitted to n storage (not shown) for exampleimplementing a database, so that the output data is stored in thestorage. In some examples, the output data may be transmitted to adisplay device (not shown) to allow a user to review the output data,for example as described above with reference to FIG. 5 or 7 . In someexamples, the output data may be stored, alternatively or additionally,in the memory device 994.

The apparatus 990 may be implemented as a processing system and/or acomputer. It will be appreciated that the methods according to any oneof the examples described above with reference to FIGS. 1 to 7 arecomputer implemented methods, and that these methods may be implementedby the apparatus 990.

FIG. 10 shows another example for possible first and/or secondpredefined patterns. Specifically, the first predefined pattern may be asampling grid 1001 comprising a plurality of grid points 1002. The gridpoints 1002 are distributed in first or second medical imaging data 220,330. The grid points 1002 are located relative to either the firstlocation 224 or the candidate second locations 334, 340, 440. For thesake of easy reference, the first location 224 or the candidate secondlocations 334, 340, 440 are together referred to as given location inthe following. In particular, the sampling grid 1001 may be centeredwith respect to the given location, with the given location sitting inthe center of the sampling grid 1001.

The sampling grid 1001 may be a regular grid which has one or more axisor planes of symmetry. The planes or axis of symmetry may run throughthe respective given location. Further one or more axis or planes ofsymmetry may intersect in the respective given location. According tosome examples, the sampling grid 1001 respectively fully spans therespective imaging spaces of the first and second medical imaging data.According to some examples, the sampling grid 1001 may betwo-dimensional. According to other examples, the sampling grid 1001 maybe three-dimensional.

Each sampling point 1002 of the sampling grid 1001 may correspond to anentry of a vector, the vector being the descriptor for the givenlocation. Thereby, each entry may be based on the values of elements ofthe underlying image data located in the vicinity of the respectivesampling point 1002. Thus, in other words, the process may comprisecalculating a first descriptor based on the sampling grid 1001 appliedto first medical imaging data 220 with respect to the first location224. This first descriptor is then compared to a plurality of seconddescriptors of a plurality of candidate second locations 334, 340, 440in a second medical imaging data set 220 as explained before. Thereby,the second descriptors may respectively be obtained by applying thesampling grid 1001 to the second medical imaging data 330 relative tothe respective candidate second location 334, 340, 440. Self-speakingthis way of descriptor-extraction may also be combined with otheraspects as explained above, e.g., the hierarchical refinement introducedin connection with FIG. 4 and the third candidate locations 448.

As shown in FIG. 10 , the sampling points 1002 according to someexamples are not equidistant, but the average distance between thesampling points 1002 increases with increasing distance to the givenlocation. That is, the density of the sampling points per unit area orvolume decreases with increasing distance to the given location.

Further, each sampling point 1002 may be conceived as a center point ornode of an associated sampling area 1003 defined around the respectivesampling point 1002. In a way, the sampling grid 1001 may be seen ascomprising a regular pattern of sampling areas 1003 which respectivelycover the first or second medical imaging data sets 220, 330 if appliedthereon. Sampling areas 1003 of neighboring sampling points may share acommon border or margin (which may be a line or plane).

If the distance between individual sampling points 1002 gets larger withincreasing distance to the given location, this has the consequence thatalso the sampling areas get larger with increasing distance to the givenlocation.

According to some examples, the descriptor for any given location (e.g.,the first location, or the candidate second locations, or the candidatethird locations) is a vector comprising a plurality of entries, eachentry being representative of the values of the elements locatedrelative to a respective one of the plurality of grid points 1002.Specifically, an entry may be based on an average property or feature ofelements comprised in the respective sampling area 1003. Specifically,the average may be based on the values of the elements. This isexemplified in FIGS. 11 and 12 .

In embodiments where the spacing between individual sampling points 1002increases with increasing distance from the given location as in FIGS.10 to 12 , elements in the vicinity of the given location contribute tothe descriptor to a greater extent than elements which are more distant.This is because the contributions of the latter are increasinglyaveraged out.

The inventors have recognized that the usage of sampling grids fordescriptor extraction constitutes a fast and at the same time veryaccurate way of encoding image information for feature matching. Inparticular, the usage of sampling grids 1001 spanning the entirerelevant image space is computationally fast. The decrease in samplingpoint density may make sure that more relevant elements of a medicalimaging data set contribute more than others.

According to some examples, the sampling grids 1001 are defined based ongeneralized coordinates or world coordinates to take into account thefact that pixel spacings could be different in different studies.Therefore, descriptor voxel value offsets may be calculated for eachvolumetric image accounting for pixel spacings and specified scalingfactors for the selected refinement level. Once the offset coordinatesare computed, the descriptor may be based on pixel intensity values onthose locations. Similar anatomical locations would produce similardescriptors with this approach.

According to some examples, a number of different predetermined samplinggrids 1001 may be provided. These sampling grids 1001 may be differentin terms of the number and/or density of sampling points 1002. Thesampling grids 1001 may be suited for different use cases and/orcontents and/or types of medical imaging data sets. For instance, theremay be dedicated sampling grids 1001 for MRI medical imaging data setsand different dedicated sampling grids 1001 for CT medical imaging datasets. Further, there may be dedicated sampling grids for different bodyregions and or organs shown in the medical imaging data. Optionally, thesampling grid 1001 to be used for matching locations in first and secondmedical imaging data may be selected from the plurality of predeterminedsampling grids 1001 based on the first and/or second medical imagingdata. This may involve determining a type and or a content of the firstand/or second medical imaging data and select the sampling grid 1001from the plurality of sampling grids 1001 on that basis. With that, anoptimized sampling grid 1001 may be selected for the respective usecase.

As an alternative, the sampling grid 1001 may be generated/adaptedspecifically for the respective image data. According to some examples,this may be carried out by a trained algorithm trained to generate(predefine) a sampling grid 1001 for the ensuing descriptor extraction.

In FIG. 13 , a schematic flow diagram depicting optional method steps isshown. FIG. 13 is directed to a workflow for providing a starting pointfor the iterative search for the second location and/or verifyingresults obtained by way of finding similar descriptors as hereindescribed.

In a first step S13-10, an image registration between the first andsecond medical imaging data is determined.

Providing at least one image registration, according to some examples,may in general comprise registering a target image (e.g., the firstmedical imaging data 220) with a reference image (e.g., the secondmedical imaging data 330). According to some examples, this may compriseobtaining a transformation function between target and reference imagedata that determines a relationship between the coordinate systems ofthe target image data and the reference image data such that eachphysiological location in the target image is mapped to the samephysiological location in the reference image and vice versa. Thus, thetransformation may comprise a plurality of individual displacementvectors respectively associated with the pixels or voxels (i.e., theelements) of the target image and the reference image.

According to some examples, the registration may comprise a rigidregistration. A rigid registration may comprise a registration in whichthe coordinates of pixels or voxels in one image data are subject torotation and translation in order to register the image data to anotherimage data. According to some examples, the registration may comprise anaffine registration. An affine registration may comprise a registrationin which the coordinates of data points in one image are subject torotation, translation, scaling and/or shearing in order to register theimage to another image. Thus, a rigid registration may be considered tobe a particular type of affine registration. According to some examples,the registration may comprise a non-rigid registration. A non-rigidregistration may provide different displacements for each pixel or voxelof the image data to be registered and can, for example, use non-lineartransformations, in which the coordinates of pixels in one image aresubject to flexible deformations in order to register the image toanother image. Non-linear transformations may, according to someexamples, be defined using vector fields such as warp fields, or otherfields or functions, defining an individual displacement for eachpixel/voxel in an image. Rigid image registration is very effective incases when no deformations are expected. In comparison to rigid imageregistration, non-rigid image registration has a significantly greaterflexibility, as non-rigid image registrations can manage localdistortions between two image data sets but can be more complex tohandle.

In a second step S13-20, an estimated location of the given feature inthe second medical imaging data 330 is determined based on theregistration. Specifically, the first location 224 may be transformedinto the coordinate space of the second medical imaging data 330 bysubjecting it to the coordinate transformation obtained with theregistration.

The ensuing steps S13-30, S13-40, S13-50 are directed to the usage ofthe registration in the location matching process. They are optional andmay also be combined.

At step S13-30, the estimated location may be used as candidate secondlocation. In other words, the registration is used to provide an“educated guess” for the second location in order to reduce the numberof iterations required to find the selected second location.

At step S13-40, the estimated location may be used to verify theselected candidate second location. In particular, the distance betweenthe estimated location and the selected candidate second location may bechecked. If the selected candidate second location and the estimatedlocation are too far apart, this may be an indication that thedetermination of the selected candidate second location via thediscriminator extraction failed. In this regard, it should be noted thatthe registration is oftentimes more robust but may lead to less accurateresults as compared to the feature extraction. Accordingly, theregistration may allow for a good sanity check.

Furthermore, the registration may be used as a fallback option. This isexploited at step S13-50, where the estimated location may be used asthe selected candidate second location or, respectively, as the locationat which the given feature is represented in the second medical imagingdata 330, e.g., if it is determined at step S13-40 that the selectedcandidate second location determined by the descriptor matching cannotbe quite right.

A further way of optionally performing a sanity check is depicted inFIG. 14 .

At a first step S14-10, a verification descriptor for each of aplurality of candidate verification locations in the first medicalimaging data is obtained. Thereby, each verification descriptor isrepresentative of values of elements of the first medical imaging datalocated relative to the respective candidate verification locationaccording to the first predefined pattern. The predefined pattern andthe descriptor extraction may be of the same types as explained inconnection with FIGS. 1 to 12 .

At a second step S14-20 for each of the plurality of candidateverification locations, a similarity metric is calculated, thesimilarity metric indicating a degree of similarity between the seconddescriptor of the selected candidate second location and theverification descriptor for the candidate verification location.Thereby, essentially the same procedures may be applied as explained inconnection with FIGS. 1 to 12 .

At a third step S14-30, a candidate verification location is selectedfrom among the plurality of candidate verification locations based onthe calculated similarity metrics.

At a fourth step S14-40, the quality of the location determination inthe second medical imaging data is determined based on a comparisonbetween the selected candidate verification location and the firstlocation. If the determination of the selected candidate second locationwas sound, the selected candidate verification location and the firstlocation should approximately lie at the same spot. If the selectedcandidate verification location and the first location are too farapart, this may be an indication that the location matching via thediscriminator extraction failed for the particular first and secondmedical imaging data. In this case, a registration may be performed andused as a fallback as described in connection with FIG. 13 .

In FIG. 15 , a workflow is shown in which the location matching asherein described is used indirectly for determining like image regionsin two medical imaging data sets.

At step S15-10, a first region of interest in the first medical imagingdata 220 is determined. The first region of interest may be seen as theimage portion for which a corresponding portion is to be identified inthe second medical imaging data 330. The first region of interest may,in particular, be an image region the image data of which is not suitedfor the location matching based on extracting descriptors as hereindescribed. This may be the case if the image pattern in the first regionof interest does not have a sufficient discriminative strength to, e.g.,sufficiently stand out against the background. For instance, this mayhappen if the first region of interest has a rather uniform appearancewithout any remarkable features.

The first region of interest may represent an area within the firstmedical imaging data 220, which is of specific interest for the useranalyzing the first medical imaging data 220. The first region ofinterest may be a part of the first medical imaging data 220. As such,the first region of interest may have an arbitrary shape, preferably theregion of interest is of circular or quadratic or box-like form. In anycase, a first region of interest may be understood as a group of imageelements like pixels or voxels within the first medical imaging data220.

The first region of interest may be defined by a user orsemi-automatically or (fully-) automatically by the computer-implementedmethod. Thus, obtaining the first region of interest may be based onprocessing one or more user inputs to designate the first region ofinterest in the first medical imaging data 220. For instance, such userinputs may comprise scrolling to a target slice and/or defining a regionof interest in the target slice.

Further, obtaining the first region of interest may compriseautomatically identifying an anatomical feature in the first medicalimaging data 220 wherein the anatomical feature is indicative of apathological condition of a patient. In particular, this may involveapplying a detection function configured to identify anatomical featuresin medical imaging data.

At step S15-20, the first location is selected. In particular, the firstlocation may be at a different location than the first region ofinterest. The first location may relate to a given feature in the firstmedical imaging data 220 which can be easily recognized in both thefirst and second medical imaging data. In other words, the firstlocation may be automatically selected so as to have a gooddiscriminative strength in the first and/or the second medical imagingdata. For instance, the selection may be related to a particularlybright image area or a particular image pattern.

At step S15-30, an offset between the first region of interest and thefirst location is obtained. Optionally, this offset is provided ingeneralized or world coordinates that are equally applicable to thefirst and the second medical imaging data.

At step S15-40, the selected second candidate location is obtained asdescribed above. In other words, a location is identified in the secondmedical imaging data 330 that represents the given feature of the firstlocation.

At step S15-50, a second region of interest in the second medicalimaging data 330 is determined based on the selected candidate secondlocation and the offset. With that, a region is retrieved in the secondmedical imaging data 330 that corresponds to the first region ofinterest in the first medical imaging data 220. The second region ofinterest is part of the second medical imaging data 330. The secondregion of interest may have the same shape as the first region ofinterest. A second region of interest may be understood as a group ofimage elements like pixels or voxels within the second medical imagingdata 330.

The above examples are to be understood as illustrative examples of theinvention. It is to be understood that any feature described in relationto any one example may be used alone, or in combination with otherfeatures described, and may also be used in combination with one or morefeatures of any other of the examples, or any combination of any otherof the examples. Furthermore, equivalents and modifications notdescribed above may also be employed without departing from the scope ofthe invention, which is defined in the accompanying claims.

1. A computer implemented method of determining a location at which agiven feature is represented in medical imaging data, the medicalimaging data comprising an array of elements each having a value, themethod comprising: obtaining a first descriptor for a first location infirst medical imaging data, wherein the first location is the locationwithin the first medical imaging data at which the given feature isrepresented, wherein the first descriptor is representative of values ofelements of the first medical imaging data located relative to the firstlocation according to a first predefined pattern; obtaining a seconddescriptor for each of a plurality of candidate second locations insecond medical imaging data, wherein each second descriptor isrepresentative of values of elements of the second medical imaging datalocated relative to the respective candidate second location accordingto the first predefined pattern; calculating, for each of the pluralityof candidate second locations, a similarity metric indicating a degreeof similarity between the first descriptor and the second descriptor forthe candidate second location; selecting a candidate second locationfrom among the plurality of candidate second locations based on thecalculated similarity metrics; and determining the location at which thegiven feature is represented in the second medical imaging data based onthe selected candidate second location.
 2. The computer implementedmethod according to claim 1, wherein the method further comprises:generating output data indicating the determined location at which thegiven feature is represented in the second medical imaging data.
 3. Thecomputer implemented method according to claim 2, wherein the methodfurther comprises: transmitting the output data to a display device todisplay a representation of the second medical imaging data and anindicator indicating, on the representation of the second medicalimaging data, the determined location at which the given feature isrepresented.
 4. The computer implemented method according to claim 1,wherein determining the location at which the given feature isrepresented comprises determining, as the location at which the givenfeature is represented in the second medical imaging data, the selectedcandidate second location.
 5. The computer implemented method accordingto claim 1, wherein each candidate second location is a location atwhich a respective previously detected feature is represented in thesecond medical imaging data.
 6. The computer implemented methodaccording to claim 5, wherein the first descriptor for the firstlocation in the first medical imaging data is obtained for each of aplurality of said given features and wherein the method comprises:calculating said similarity metric between the first and seconddescriptors for pairs of the first locations and candidate secondlocations; assigning each first location a different one of thecandidate second locations based on the calculated similarity metricsbetween the pairs of the first locations and candidate second locations;and determining, for each said first location, as the location at whichthe respective said given feature is represented in the second medicalimaging data, the second candidate location assigned to the firstlocation.
 7. The computer implemented method according to claim 1,wherein the candidate second locations are locations distributed throughthe second medical imaging data in a second predefined pattern.
 8. Thecomputer implemented method according to claim 7, wherein determiningthe location at which the given feature is represented in the secondmedical imaging data comprises: determining, based on the selectedcandidate second location, a plurality of candidate third locations inthe second medical imaging data, obtaining a third descriptor for eachof the plurality of candidate third locations in the second medicalimaging data, each third descriptor being representative of values ofelements of the second medical imaging data located relative to therespective candidate third location according to the first predefinedpattern; calculating, for each of the plurality of candidate thirdlocations, a similarity metric indicating a degree of similarity betweenthe first descriptor and the third descriptor for the candidate thirdlocation; selecting a candidate third location from among the pluralityof candidate third locations based on the calculated similarity metricsindicating the degree of similarity between the first descriptor and therespective third descriptors; and determining the location at which thegiven feature is represented in the second medical imaging data based onthe selected candidate third location.
 9. The computer implementedmethod according to claim 8, wherein the candidate third locations arelocations distributed through the second medical imaging data in a thirdpredefined pattern in a region of the selected second location, whereina distance between the candidate third locations in the third predefinedpattern is less than a distance between the candidate second locationsin the second predefined pattern.
 10. The computer implemented methodaccording to claim 1, wherein a descriptor for a given location is avector comprising a plurality of entries, each entry beingrepresentative of the values of the elements located within a respectiveone of a plurality of predefined boxes located relative to the givenlocation according to the first pattern.
 11. The computer implementedmethod according to claim 10, wherein each entry of the plurality ofentries of the vector is an average of the values of elements locatedwithin a respective one of the plurality of predefined boxes.
 12. Thecomputer implemented method according to claim 1, wherein the firstpredefined pattern is a sampling grid comprising a plurality of gridpoints located relative to a given location, and a descriptor for thegiven location is a vector comprising a plurality of entries, whereineach entry is representative of the values of the elements locatedrelative to a respective one of the plurality of grid points.
 13. Thecomputer implemented method according to claim 12, wherein a density ofgrid points per elements of first and second medical imaging datadecreases with increasing distance to the given location.
 14. Thecomputer implemented method according to claim 12, wherein each entry ofthe plurality of entries of the vector is an average of the values ofelements located within a sampling area respectively defined around arespective one of the plurality of sampling points.
 15. The computerimplemented method according to claim 1, further comprising: obtaining afirst region of interest in the first medical imaging data; determiningan offset between the first region of interest and the first location;and determining a second region of interest in the second medicalimaging data based on the location and the location and the offset. 16.The computer implemented method according to claim 1, wherein themedical imaging data is integral image data.
 17. The computerimplemented method according to claim 1, wherein the similarity metriccomprises a normalized mutual information similarity between the firstdescriptor and the second descriptor.
 18. An apparatus, comprising: anon-transitory memory device for storing computer readable program code;and a processor in communication with the memory device, the processorbeing operative with the computer readable program code to performoperations for determining a location at which a given feature isrepresented in medical imaging data, the medical imaging data comprisingan array of elements each having a value, the operations includingobtaining a first descriptor for a first location in first medicalimaging data, wherein the first location is the location within thefirst medical imaging data at which the given feature is represented,wherein the first descriptor is representative of values of elements ofthe first medical imaging data located relative to the first locationaccording to a first predefined pattern; obtaining a second descriptorfor each of a plurality of candidate second locations in second medicalimaging data, wherein each second descriptor is representative of valuesof elements of the second medical imaging data located relative to therespective candidate second location according to the first predefinedpattern; calculating, for each of the plurality of candidate secondlocations, a similarity metric indicating a degree of similarity betweenthe first descriptor and the second descriptor for the candidate secondlocation; selecting a candidate second location from among the pluralityof candidate second locations based on the calculated similaritymetrics; and determining the location at which the given feature isrepresented in the second medical imaging data based on the selectedcandidate second location.
 19. The apparatus of claim 18, wherein theprocessor is operative with the computer readable program code togenerate output data indicating the determined location at which thegiven feature is represented in the second medical imaging data.
 20. Oneor more non-transitory computer-readable media embodying instructionsexecutable by machine to perform operations for determining a locationat which a given feature is represented in medical imaging data, themedical imaging data comprising an array of elements each having avalue, the operations comprising: obtaining a first descriptor for afirst location in first medical imaging data, wherein the first locationis the location within the first medical imaging data at which the givenfeature is represented, wherein the first descriptor is representativeof values of elements of the first medical imaging data located relativeto the first location according to a first predefined pattern; obtaininga second descriptor for each of a plurality of candidate secondlocations in second medical imaging data, wherein each second descriptoris representative of values of elements of the second medical imagingdata located relative to the respective candidate second locationaccording to the first predefined pattern; calculating, for each of theplurality of candidate second locations, a similarity metric indicatinga degree of similarity between the first descriptor and the seconddescriptor for the candidate second location; selecting a candidatesecond location from among the plurality of candidate second locationsbased on the calculated similarity metrics; and determining the locationat which the given feature is represented in the second medical imagingdata based on the selected candidate second location.